Context / original problem

In our deployment, we have recently started seeing quetz crashes with the following error message:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

This struck me as odd because we configure the pool / overflow sizes manually via the quetz settings / environment variables. The values in the error message neither match our settings nor the quetz default settings, indicating that both sets of settings are ignored.

Diagnosis

I have traced this to the fact that our main entrypoints for connecting to the database (get_engine and get_session) are used inconsistently in the code base. Effectively, it is currently up to the caller to make sure that the right settings are propagated from the configuration to the kwargs for these functions, which often does not happen. Therefore, some connections are configured correctly, some are not.

While looking through the code, I also noticed that we also treat session management inconsistently throughout the code base. Sometimes we protect sessions via a contextmanager that closes the session after use, sometimes we do it manually, sometimes we never close the session.

Proposed solution / changes

• Bumped sqlalchemy to version 2. In sqlalchemy 2, Session objects offer the context manager protocol and automatically handle cleanup.
• Refactored all session obtaining code to go through a single function: get_session(config: Config | None) -> Session. By default, if config is not provided, get_session will use the default global Config object. Sessions obtained in this way are guaranteed to respect the config settings. It is now easy to grok if sessions are handled correctly: If you see with get_session(...) as db:, everything is good. If you see get_session used in any other way, it's bad.
• Added type hints in / near code I touched